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Abstract 

The challenging of assessing formal and informal online learning at scale includes various issues. Many 
universities who are now promoting “Massive Online Open Courses” (MOOC), for instance, focus on relatively 
informal assessment of participant competence, which is not highly ‘quality assured’. This paper reports best 
practices on the use of a web videoconferencing application to quality control student assignments through 
online oral examination at scale. In this case study, we examine the use of a simple online conferencing 
technology FlashMeeting (FM) by a Brazilian University to provide ‘quality assurance’ in the assessment of 
twelve online postgraduate courses in Law for 20,000 students. Our research questions investigate the 
benefits and recommendations of using FM in online oral exams at scale. Our qualitative and quantitative 
data analysis centres on 3,462 short format interviews through FM conducted for this purpose by a group of 
around fifty assessors from September 2008 to September 2012. The effective use of FM provided evidence 
with respect to high quality assurance recognised by the Institution with respect to: students’ identity, their 
knowledge and ownership of written work. The key benefits identified from the perspective of assessors and 
students were: reliable examination, credible technology, authentic assessment, interactive e-Viva, low cost, 
scalable process and practical testing in terms of time, effort and money. 

Keywords: Assessment at scale; quality assurance; videoconferencing; Massive Online Open Courses; 
Higher Education; oral online exams; e-Viva 

Introduction 

The issue of quality assuring the work of a remote student has been a long-standing challenge 
to Distance and Open Universities (Gaytan, 2005; Robles & Braathen, 2002). Most of these 
organizations still place an emphasis on summative exams in a controlled physical proctored location 
(Hollister & Berenson, 2009; Harmon & Lambrinos, 2008; Wynne & Lopes, 2006). A human proctor 
in a physical exam centre manually checks the student identity paperwork before the student is 
invited to complete a short time length ‘hidden paper’ examination. Many proctoring centres and 
technologies are now commercially available. Their aim is to add a quality assurance element in 
‘observing and recording’ the student during the exam on a ‘locked down’ computer via a webcam 
to reduce the temptation to cheat during the hidden paper test. 

The examination paper is presented in this way because it can be guaranteed to be the student’s 
work due to the physical constraints of the exam venue. The venue may be in many different world 
locations, the exam may even be on a ‘locked down’ networked computer to allow rapid submission 
of the work, but the management of the process would be recognizable to someone examining two 
centuries ago. However, the assessment challenge has now broadened to many more organizations 
in conventional universities who are now promoting “Massive Online Open Courses” (MOOC) online. 
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These are aimed at a very large-scale student learning experience, and currently focus on relatively 
informal assessment of participant competence, which is not highly ‘quality assured’ such as a 
‘badge’ based (Cross, Whitelock & Galley, 2014) on a multiple choice quiz (Perna et al., 2014). As 
MOOC provision matures, the issue of providing ‘high stakes’ examining for the learner experience 
(Koller, Ng, Chuong & Chen, 2013) is gaining focus—in particular because the certification component 
of a MOOC offers a possible business-model driven by “for fee” element in an otherwise free and 
open system. 

The value of any certificate must rest on a number of quality assurance issues: Did the student 
actually attend the course, and contribute? Did the student submit assignments and answer questions 
alone or with the assistance of others? Is the student able to demonstrate the competences that 
the course aims to teach? A range of work has sought to ensure student compliance with the course 
structure, from simple systems that require a student to manually check the completion of each 
‘course page’ in the online learning environment, to research into systems, which require students 
to constantly confirm their attention during the replay of recorded lectures (Canessa, Tenze & 
Salvatori, 2013). Of course, ensuring that a student has been exposed to the offered teaching 
material does not assure that anything has been learned. At higher levels, certainly, a preferred 
technique is to use ‘low stakes’ continuous assessment through the teaching. Short answers or 
multi-choice questions, for instance, allow the quicker student to skip content they find easy whilst 
still confirming their competence as they move through the material. In all such cases, ‘raising the 
stakes’ in e-Assessment is clearly a challenging topic in education both through the more effective 
use of technology and in re-thinking the nature of the examination itself. 

The use of remote videoconferencing to reduce the cost and complexity of face-to-face physical 
meetings is understandably popular. Indeed, webcam videoconferencing technology is now very 
common for many meeting formats, with a wide range of robust technical systems such as Skype™ 
and Apple’s FaceTime™ and with cameras now commonly built into many computers, tablets and 
phones. In contrast, for events which are considered “high stakes”, i.e. where the process of the 
meeting can be complex and the outcome is very significant to participants, the replacement of 
the physical with the virtual has been slower. For ‘high stakes’ outcomes such as job interviews, 
the potential has been explored as the technology has matured, and early reservations (Kroeck & 
Magnusen, 1997) are being overcome. For example, a compelling case shows that short online 
video interviews are just as effective in student selection for a graduate entry programme as the 
much more expensive, face-to-face equivalent (Tiller et al., 2013). However, whilst video interviews 
are now in use for testing and validating performance effectively, for instance in language exams, 
users are still reporting concerns over increased stress of the online performance, over how they 
might feel about face-to-face assessment, and concerns about the technical set up, use of ‘recording’ 
features and need for effective technical support for the technology (Kim & Craig, 2012). In all these 
examples, there are still concerns about the quality assurance of the online event relative to the 
physical, and the different pressures it may place upon the participants. 

The face-to-face viva voce examination remains a ‘gold standard’ for testing a student’s knowledge 
of a body of work. It is particularly popular in higher degree programs in which the examiner can 
spend time to interrogate a student with respect to a written submission they have made, often a 
thesis or dissertation. One valuable attribute of an oral examination, in combination with a written 
submission, is that it can be used by an expert examiner to both probe the students’ knowledge of 
topics which were not covered well in the written submission, but most especially that it offers a 
quality assurance opportunity to test the students ‘ownership’ of the submitted work, and to reduce 
the risk that the claimed work is actually that of another. 
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As a high-quality procedure, the oral examination is typically reserved for ‘very high stakes’ 
examining due to its cost and complexity to organize and manage—requiring a physical meeting of 
the student and the expert examiner, plus often the need to host an observer (to witness the process 
and to accommodate any ‘appeal’ claim that may later arise). In particular, because the examiner 
is scheduled with one student at a time, this generally rules it out of any large-scale process due 
to the expense of the examiner’s time and physical logistics of meeting with each student to discuss 
their work. If a model of videoconferencing use could be applied to bring the gold standard to a 
wider use at scale that might be very compelling. 

In this study, a significant number of online oral examinations are analysed, including quantitative 
data provided by the FM web conferencing application as well as qualitative feedback from students, 
examiners, supervisors and course coordinator. Our investigation centres on “how can we assure 
the quality of a large-scale, low cost and flexible distance education online assessment?” The 
objectives of this investigation are to discuss not only advantages and challenges of oral exams 
through a web videoconference application, but also suggest recommendations grounded on best 
practices. This investigation therefore aims to address these questions: 

• Can online videoconferencing technology be effectively used as a quality assurance mecha¬ 
nism in a scaled, low cost and flexible way for distance education? What are the requirements 
and suggestions? 

• What are the key benefits and success factors in the effective deployment of online video 
supported viva-voce examinations? 

• What would be the expected barriers and challenges to QA in a distance assessment context? 

Research context: case study in Brazil 

The case study focuses on the education network LFG, ‘Luiz Flavio Gomes’, which has partnered 
with UNISUL, Universidade do Sul de Santa Catarina, for providing the following online postgraduate 
programmes in Law: 

1. Constitutional Law 

2. Public Law 

3. Civil Procedural Law 

4. Social and Practical Function of Law: Public Law 

5. Social and Practical Function of Law: Tax Law 

6. Criminal Sciences 

7. Tax Law 

8. Procedural Law: Great Transformations 

9. Notary and Registry Law 

10. Social Function of Law: Procedures, Constitution and New Rights 

11. Election Law 

12. Constitutional Limits of Investigation in Brazil 

The agreement between UNISUL and LFG was in force from 2005 to 2008. More than 20,000 
students registered for the Law programmes offered. In compliance with the rules and regulations 
of the State Council of Education of Santa Catarina and the National Council of Education, vivas 
were initially optional and later compulsory for two of the programmes, namely Public Law and Civil 
Procedural Law. 

The certificates of students who opted for a viva bore a reference to the rules and regulations of 
the State and of the National Council of Education. For those who chose not to have a viva, these 
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certificates bore a reference to the rules and regulations of the State Council of Education only. The 
certificates of the students enrolled in the programmes which required a viva bore a reference to 
the rules and regulations of both the State and the National Council of Education and those students 
who did not show up for the compulsory viva were not approved and, therefore, were not issued 
with a certificate. 

Every semester, registration was open to each of the above-mentioned programmes with a 
minimum requirement of 100 students registered per programme. In some cases, the number of 
students registered per programme was as high as 1,600. The duration of these programmes was 
one year and a half. 

The pedagogical methodology adopted by LFG-UNISULfocused on online-based and telepresence 
courses grounded on four key features: synchronous and asynchronous content delivery, continual 
online learning support, written and oral e-assessment as well as two possibilities of certification, 
namely Training for the Job Market (with 360 contact hours) and Higher Education Teacher Training 
(with 450 contact hours). The two modalities shared the same curriculum and the Higher Education 
Teacher Training Certificate had two additional courses each with 45 contact-hours: 

1) Research Methodology; and 

2) Higher Education Methodology. 

Course content 

A team of external professors and professional specialists in Law was responsible for running 
telepresence lectures once a week as well as developing the learning content. Students would 
participate in the tele-lectures at the regional centre located closest to their work or home. They 
were also registered in the virtual environment denominated EVA (Espago Virtual de Aprendizagem ) 
—Learning Virtual Space, of UnisulVirtual. In the EVA, they could access learning materials as well 
as group activities, optional individual exercises and discussion forums. 

Learning support 

The UnisulVirtual team responsible for these programmes consisted of 40 lecturers, 7 programme 
co-ordinators, 14 student support staff and 30 clerical and technical staff. Learning support was 
offered by the lecturers in the EVA and other kinds of support were provided by the Student Support 
Team either in the EVA or by telephone. Pedagogical support was also offered during the learning 
activities in the EVA as well as during the supervision and assessment of students’ final papers at 
the end of the programme. 

E-assessment 

The key aspects for assessing students were quality of learning outcomes, authenticity and tele¬ 
lecture attendance control. The assessment system was based on online assignments and/or exams 
as well as a final exam through web videoconferencing based on safety precautions for ensuring 
the reliability and credibility of results. E-assessment for each course included all activities available 
in the EVA (including a distance exam), and a face-to-face exam. At the end of the programme, 
students had to produce a final paper—with its oral defence through an online viva required by only 
two of the programmes: Public Law and Procedural Law: Great Transformations. The lecturers were 
responsible for mediating the teaching and learning processes and also for motivating students and 
creating learning opportunities. They could be reached through the EVA to provide students with 
guidance during the activities of each course or module. They were also in charge of formulating 
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and grading exams. The Student Support Team was responsible for giving students technical, 
administrative and, occasionally, motivational support. 

The online viva process involved a group of fifty three lecturer-examiners from UNISUL. The 
examiners were provided with a ‘pre assessed’ final paper for each student, and were briefed that 
their primary purpose was to confirm that the paper was the work of the submitting student. The 
paper itself was to have been already graded by the final paper supervisor before the date of the 
scheduled oral exam. Examiners should only confirm students’ identity and probe their knowledge 
of the submitted work to ‘quality assure’ the submission. Furthermore, each session was ‘auto 
recorded’ to a unique coded URL hosted on the event server (this being a default feature of the FM 
service). The recording was deemed to be ‘co-owned’ by both the student and the institution and 
was not to be edited by either party. Ergo, the student was at liberty to review her/his performance 
at any time and indeed to share it with family or friends at their discretion. Equally, the institution 
could use the recording to confirm and cross-check any unresolved identity issues, but also the 
student could use the recording as evidence in any appeal. 

Certification 

Students were issued with a Postgraduate Certificate provided that: they had attended at least 75 
percent of the tele-lectures in the regional centres; they had obtained no grade under “C” in any of 
the courses during the programme; their final papers had obtained a passing grade; and they had 
obtained a passing grade after the viva in the case of the two programmes which required an oral 
defence of the final paper. The weightings of the exams held for each course were as follows: 
distance exams/assignments/VLE activities (40 percent); face-to-face exam (60 percent). 

Vivas were conducted via the web videoconferencing tool called FlashMeeting (FM), developed 
by KMI—the Knowledge Media Institute of the Open University UK—and could last no longer than 
30 minutes. Students’ presentations of theirfinal papers could last up to 20 minutes and the remaining 
minutes could be used for answering questions if necessary. The requirements for the oral 
presentation were a clear identification of the student through a webcam as well as clear sound 
during their presentation for the examiners. Students were told that the presentations were recorded 
and that they would be accessible at any time. In case of fraud the FM replay can be used as a 
proof. The student received the URL of their presentation and they were allowed to share them with 
their colleagues and family. In order to be approved, students must be able to show the knowledge 
presented in their written papers. 

The viva was compulsory for the programs of Public Law and Procedural Law but optional for the 
other programmes. 

The UNISUL staff confirmed that students were carefully instructed with respect to the objectives 
of the e-Viva, for example, to: 

• Confirm their authorship of the written assessed work under consideration; 

• Prove that their identity in the e-Viva was a match with the name on the written work; 

• Demonstrate that they know the content of the submitted work. 

Students and staff confirmed that they understood the requirements to succeed in the e-Viva: 
(a) that they could use the minimum infrastructure—webcam and audio; (b) that they would be 
expected to interact with the examiner, answering questions during a 30 minute ‘booked’ online 
event; (c) that the minimum ‘acceptable interaction’ time would be 15 minutes, of the 30 minute 
slot—to allow for any technical issues; (d) that the assessors should be objective and the students 
should provide complete answers; (e) and that any participant could attend the e-Viva from any 
location, e.g. from home, a local ‘LAN house’ or at a Learning Centre. 
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Research design: approach and technology 

This case study is based on qualitative and quantitative data analysis of 3,462 short format interviews 
through the FM web videoconferencing technology conducted for this purpose by a group of around 
40 assessors from September 2008 to September 2012. A total of 3,205 vivas were held which 
included rebooking of events, out of which 1,300 were compulsory and 1,905 were optional during 
the period of September 2008 to September 2012. 

FM is a web video conferencing system, which has come from the FlashMeeting project 
(http://flashmeeting.com) hosted at the UK’s Open University (Scott, Tomadaki & Quick, 2007). FM 
events are fixed in time, and therefore must be booked. The system generates a unique web URL 
for each event, which acts as a key to the live meeting, when circulated to meeting attendees, who 
simply click on the link to gain access to the videoconference. Attendees do not need to be registered 
in the system or have an account to participate. The controls to the system are very simple, and 
typically FM does not require any additional downloads to an up-to-date web browser. Generally, 
events are set to “auto record” on the server, and those recordings are available via the same URL 
as the live event, so that a participant in the live meeting already has ‘the key’ to the recording. 
FM recordings can be edited, annotated and syndicated—but these features were not used in this 
scenario. The application provides a ‘push-to-talk’, simple audio/video broadcasting which allows 
only one person to speak at any one time; with a hand-up to talk model to speak in which the current 
‘speaker’ chooses when to hand over to any requesting user. There is a parallel text chat channel 
in which any participant may type during the event. 




Figure la: the “live view” of an FM e-Viva Figure 1b: the “recorded view” of an 

event with a chat area. e-Viva event with its timeline. 

Figure 1 presents two screenshot views of an e-Viva event which was conducted 11th March 2011. 
They show a view as the participants would have seen it live (figure la) and how the same event 
would appear as a recording (figure 1b). (Note the images of participants have been replaced and 
the names obscured with ‘xxxx’ to preserve anonymity; but all other features remain unedited). In 
figure la we can see that the examiner (here designated as Prof Dxxx) is addressing the student, 
and the student (designated Rxxx) has raised a hand to respond. Figure la is presented from the 
student’s point of view, and shows that the examiner is speaking with more than twelve minutes of 
the ‘booked time’ remaining. However, she has already typed the only text chat line, which was 
used in this particular event: “aprovada”, meaning ‘you passed’, so is probably drawing this e-Viva 
to a close. In the recording (figure 1b) we can see an earlier moment in the event (elapsed time is 
2 min. 42s.’, indicated by the segment highlighted to the right of the screen, and the ‘playback line’ 
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in the bottom visualization starting to play the student’s first long answer (the horizontal bars in the 
Rxxx line being the student speaking). This event can be replayed by anyone who has the URL 
and users who are administrators in the FM can also export the meeting to an audio file, which 
might be used for transcription. 

In FlashMeeting, the metadata produced after each meeting can be used to interpret the event 
in different ways (see Figure 2). For example, these different kinds of events may contain different 
communication patterns, which can be interpreted in different sense-making representations. These 
graphs can be a timeline of the meeting visualization or the event shape graphics, which can be 
grouped by events of an online community (Okada, 2008). The IP resolutions of FlashMeeting users 
include information about the use of the tool by different communities. Using people’s IP addresses, 
their location can be plotted on a world map. In this way, maps can be generated to show the 
distribution of users for both the attendance of live FlashMeetings and FlashMeeting replay access. 
For example, the IP resolution of the users connected to the live FlashMeetings shows how the tool 
is being used to connect people from the same social network or community of practice, while the 
IP resolution of the users viewing a public replay shows the learning impact of the event reuse in 
different parts of the world (Okada, Tomadaki, Buckingham & Scott, 2008). 
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Figure 2: FM Metadata exported to an Excel file 

Based on FM Metadata (Fig. 2), the quantitative data related to all web conferences were analysed 
to identify meetings attended, meetings replayed, meeting with chat, meetings ‘duration, number of 
participants and frequency of bookings. 

Approximately 4,648 events from 2008 to 2012 were booked by the UNISUL department of oral 
examinations support. These events include 3,462 oral examinations as well as 1,186 events not 
attended or attended by only one person. All participants were aware that meetings were recorded, 
could be replayed either by students or Institution. Approximately 60 percent of meetings (2,029) 
occurred without chat, which means participants could interact well using only the broadcast audio/ 
video. In order to identify patterns and most frequent format of events, data analysis comprised the 
timelines of the shortest and longest exams with or without chat as well as with and more than two 
users. A few categories emerged, which were described in findings: short e-Viva with tests, short 
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interactive e-Viva without tests, long interactive e-Viva, long e-Viva with student’s presentation, 
meetings with three participants and administration meetings. These categories were used to select 
random sets of events for qualitative analysis of the audio transcription, which allowed the detailed 
description of these six types of events and the elaboration of qualitative instruments: a semi- 
structured questionnaire with UNISUL University and a script for open interviews led by the Open 
University. 

Therefore, qualitative analysis was sought to add insight to the quantitative data discussed above. 
In the context of formal examining, the intrusion of additional surveying was kept to a minimum; 
and it was felt that the staff experience was at least as important as the student experience of 
this process. Examined students were surveyed in two phases: firstly, a written semi-structured 
questionnaire was distributed by the UNISUL organization in 2013; secondly, the OU survey team 
contacted a small sample of students to conduct a series of open interviews. 

The questionnaire was distributed to 100 students, sampled from the latest group of e-Viva exams 
in 2011-2012 to run through this system; and for the interviews five students agreed to take part 
in the more detailed qualitative follow up. The OU team also interviewed an additional seven UNISUL 
staff members involved in the e-Viva process: the programme coordinator, the technical support 
leader and five assessors. These open interviews discussed the interviewees reflections on the 
e-Viva processes which took place in 2012 and earlier, and all interviews were conducted in the 
latter half of 2013. A few categories emerged from the qualitative analysis of semi-structured 
questionnaires and open interviews with students: feelings, preferences, suggestions for examiners 
and recommendations for other students. Other categories emerged from the examiners’ interviews: 
requirements for online exams, solutions used by students related to low internet connection and 
institutional benefits. 

Findings 

Based on quantitative data analysis of 3,462 FM web conferences, 90 percent refers to successful 
e-Viva meetings (3,147). Sixty three percent (1,990) were expected meetings whose duration varies 
from eleven to twenty minutes. While thirty seven percent (1,157) were short meetings—seven to 
ten minutes. This means that even due to any technical reason or delays, the shortest period of 
seven minutes was enough for a successful oral exam. 

There were 3,202 e-Viva events, which took place with only two participants, the student and the 
examiner. A further 143 events were conducted with an additional attendee—typically either technical 
support person or a co-supervisor in attendance. 

There were 32 coordination events for staff; and 85 technical events which took place during the 
surveyed period. Approximately 26 e-Viva events were booked per day, with two parallel meetings 
during the most popular time for participants: lunch time (11h00-14h00) and late evening (16h30- 
19h00). 

According to the UNISUL coordinator, students were not supported by any prior training on the 
use of the system and relied upon “local technical support” for any difficulties encountered. Indeed, 
approximately 70 percent of students elected to take the e-Viva in a local University approved 
Learning Centre because it was quiet, and with a good quality, guaranteed internet connection. 
None of the students had used the system previously, but the majority (90 percent) reported that 
they found the system easy to use. Students who encountered problems were able to ‘rebook’ a 
new slot (e.g. where an event, which made them late for their booked slot, or where the 30 minute 
slot was taken up with too many technical issues). 
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Common e-Viva patterns 

After selecting and analysing only the successful events (not used for testing, over ten minutes 
duration and with at least two participants), it was possible to identify six common patterns: 

(1) Short e-Viva with tests: These online meetings comprise four minutes of tests and introduc¬ 
tion, then seven minutes of questions and answers and finally less than one minute of 
communication of results. Figure 3 shows a student (Melissa) interacting with her examiner 
(Deisi). The event started at 11 min. 5s., Deisi and Melissa started testing her equipment, then 
Deisi asked two questions (Q1, Q2), Melissa replied each question during approximately two 
minutes (A1, A2). 
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Yes No Abstain Unvote Reset :-) :-< -0 ;-) Time 

Figure 3: Short e-Viva with tests 



(2) Short interactive e-Viva without tests : These online meetings comprise ten minutes of several 
questions and answers with less than one minute of communication of results. Figure 4 shows 
a student (Julio) interacting with his examiner (Fabio). The event started late at minute 20, 
but soon Fabio and Julio realised that their equipment was fine. Fabio presented six questions 
which were replied by Julio fast (with answers less than one minute). The online exam lasted 
approximately 7 minutes. 
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Figure 4: Short interactive e-Viva without tests 

(3) Long interactive e-Viva: These online meetings consist of 15 to 20 minutes with several 
questions and answers with less than one minute of communication of results. The example 
presented through Figure 5 shows a student (Leonardo) interacting with his examiner (Prof. 
Andre). The event started at minute 0, Prof. Andre presented seven questions with comments, 
which were replied by Julio with less than one minute answers. 
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Figure 5: Long interactive e-Viva 


(4) Long e-Viva with student’s presentation: These online meetings consist of 20 minutes in total 
with student’s presentation followed by questions and answers. Figure 6 shows that during 
the first two minutes, the student Daniela and examiner Prof. MSc Susana tested their 
equipment. Daniela then presented her work in ten minutes. During the next seven minutes 
of interaction, the examiner Susana asked two questions which were replied by Susana. The 
last minute the examiner presented the result and the student shared her feedback about 
the assessment. 
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Figure 6: Long e-Viva with student’s presentation followed by questions and answers 

(5) Meetings with three participants: student, supervisor and examiner: These online meetings 
were rare. Figure 7 shows the student (Marco) his supervisor (Dr. Antonio) and the examiner 
(Carlos). During the first eight minutes, the student Marco presented his work, the supervisor 
Antonio then shared comments about the written work, in which some paragraphs should be 
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Figure 7: Long e-Viva with 3 participants ( student, supervisor and examiner) 
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presented as a quote with student’s comments and interpretation. Student agreed with all 
changes. Examiner Carlos presented a few questions, which were replied by the student 
correctly. He was then approved after resubmission of his written work. 

(6) Administration Meetings These online meetings lasted one hour. Figure 8 shows the person 
responsible for booking the exam meetings (Vinicius) discussing with all the other five Units 
in Brazil for scheduling the period of FMs for each group of student. 
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Figure 8: Administration Meetings 

Outcomes from qualitative data analysis were grouped in two categories: the student and Institution 
view. 

The student view 

Students were clearly aware of the barriers inherent in this approach. For example, some interviewees 
clearly felt that this format was not entirely satisfying, and that a face-to-face alternative might have 
offered an opportunity for them to perform better. 

“I think that if the viva was F2F we would probably have more time to discuss my work, with more 
questions and more discussion in depth. I am not sure if my work was fully read and analysed by the 
examiner. I think that time for answering the questions was very short and the experience was not good 
as I expected” (Student 7). 

A few students mentioned that the test seemed to be over very quickly, and based on the questions 
they were not satisfied that the examiners had fully appreciated all their work. But most focus on 
barriers was technical—with a good internet connection a common challenge. 

“I had problems with low connection, but it was possible to complete the viva”. (Student 7) 

However, students were also acutely aware of the benefits. Most interviewees were generally 
positive about the experience and the majority felt that it was a good opportunity for them to perform, 
even relative to the face-to-face alternative. 

“I do not have a particular preference. Both modalities are fine for me. I feel that there is no difference. 
Ftowever, although I would feel confident in attending either online or F2F events, I think that most 
students might be more nervous in F2F assessments with the physical presence of the examiner 
observing the students all the time”. (Student 5) 

Some students noted that it might even be an improvement in relationto the physical meeting. 

“I definitely prefer an online viva instead of a F2F oral assessment. The reasons are the interaction with 
examiner is fast and objective. It is possible to attend the event anywhere through the Internet and 
webcam without any technical experience. I can focus on the answer to the question without looking at 
the examiner’s face and feeling worried.” (Student 6) 

In terms of feelings, students picked up on a number of issues around the virtual viva event. Students 
who felt it was a good experience reflected on the challenge of the general viva, and the need to 
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focus on their performance as already very challenging. Students were certainly aware of the 
additional technical challenges they faced in using video for interacting with their University and felt 
that this was more likely to be the cause of problems than the event format. 

“The only problem, that might be a disadvantage for online, is if there are technical problems”. 
(Student 3) 

And having quickly accepted the format potential, students were able to offer advice to others facing 
the same challenge, 

“My recommendations for students are keep calm, feel confident and study the written work before the 
assessment”. (Student 1) 

Students generally noted that “... it is normal to feel worried ...” but also noted that the online format 
allowed them a feeling of a “... less pressure and less formal ...” space to answer the questions 
posed by the examiner than the face to face context would allow. The issue of confidence was 
picked up by most interviewees, who noted that they perhaps “... felt more confident answering 
questions through web conference than face to face ...”, because they were more focused on 
replying to the question than paying attention to the examiner’s reaction. In some senses they felt 
that the physical presence of the examiner is somehow more distracting than the virtual presence, 
and would reduce their confidence. 

Students commended on the assessment’s practicality, particularly in terms of location (that they 
could participate in the meeting from home, at a learning centre or somewhere else convenient— 
some interviewees had attended the FM at an internet cafe close to their home). Students also 
mentioned flexibility, noting that in case of problems, the exam event could be quickly booked soon 
again. One student noted that this method seemed to be very efficient—providing them with a very 
fast way to complete the exam and obtain feedback. 

The institutional view 

Examiners were also aware of the objectives of the FM e-assessment as a quality control mechanism 
essential as a requirement of the Brazil Government for Distance Education postgraduate course. 
Examiners were also committed to run the e-Exam successfully, by: 

• Testing webcam and audio. 

• Allowing seven to fifteen minutes for students to talk about the content. 

• Contacting the technical support in case of problems with audio. 

Some of them tested the system. All of them mentioned that they did not receive any formal training 
in this new medium. Most of them conducted their side of the exam at the University Centre. Some 
of them attended the exams from their homes or other working places. 

Indeed, there were some problems noted by the examiners, in particular the internet unreliability 
in parts of Brazil, In addition, there were concerns raised about changing dates with very short 
notice. Updating the system was very easy, but it was inconvenient for the examiners. Other concerns 
were related to conducting the online oral exam in a noisy place, such as “LAN houses” where 
sometimes it was not easy to hear the student over the background noise, but even in these cases 
the exams were completed without any serious problems. Indeed, their experience was generally 
very positive: 

“The online viva was a successful experience in UNISUL — LFG. Online assessments are objective, with 
good outcome in a short time; therefore they are very productive time for both examiners and students. ” 
(Examiner 1) 
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The internet connection was seen as challenging, but not a significant barrier to the overall process: 

“The technology and requirements are simple, it is not necessary to have technical support for most of 
the events—the participants’ autonomy is therefore high.” (Examiner 1) 

The technology was seen as very positive with respect to supporting an efficient and reliable e-Viva 
process: 

“Even with technical problems (which were very few and most of them related to low internet connection), 
it is possible to assess students very fast. It is a safe system because it is possible to check identity and 
had simple technical requirements. It is very suitable for Distance Education Institutions.” (Examiner 2) 

Overall, the examiners focused on a very few general issues (a) that the event offered a significant 
cost saving with respect to transport, time and location; (b) that it was very flexible, as the exam 
can be provided anywhere—any time; (c) that it offered a very reliable and fast way to run the exam 
(in parallel) and (d) to offer almost instant feedback. 

Discussions and recommendations 

The literature presents a few studies on proctored testing. Some of these studies show that an 
on-line course with proctored testing was more effective in promoting learning when compared to 
an on-line course with un-proctored testing (Wellman, 2005; Picciano, 2002). Various reasons were 
identified from the students’ perception: “face validity”, reinforcement of examination process and 
clarification of questions. However, in these studies, the exams still occur in the same place for all 
students and assessors. This is not feasible for MOOCs or online assessments in countries with a 
vast territory such as Brazil. A key factor for large scale assessment programmes is the testing 
infrastructure, which includes: test development, delivery, financial services, registration and 
scheduling and information management. The infrastructure must be reliable and scalable, so that 
as the number of tests delivered increases the system is able to support this growth (Wynne & 
Lopes, 2006). 

In this work, we focused on in-depth research on the use of FM in online courses at scale for 
postgraduate certification in Law in Brazil. We have argued that the use of an effective web 
videoconferencing for proctoring online oral exams provides high quality assurance. Findings in 
this study highlighted various benefits for Institutions, examiners and students such as: reliable 
examination, credible technology, authentic assessment, interactive e-Viva, low cost, scalable 
process and practical testing in terms of time, effort and money. 

This study presents some recommendations drawn on quantitative and qualitative data for 
students, assessors and Institutions for effective use of web videoconferencing in online courses 
at scale: 

1. Institution: 

• Selecting an easy-to-use technology for online exams that does not require training and 
allows recording of the event as a way to keep evidence of the exam. 

• Providing clear instructions about the exams to students related to both written and 
oral exams: purpose, requirements, recommended venues (e.g. quiet and good web 
connection) and criteria for approval (in order to confirm identity and authorship) 

• Clarifying the minimum requirements for assessors related to quality assurance (e.g. 
successful interaction with audio, webcam and/or chat), minimum and maximum time, 
flexibility for choosing best format of the oral exam based on the circumstances (e.g. long 
or short time after delays or technical problems). 
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2. Students: 

• Being aware of the overall requirements and recommendations provided by institution as 
well as other factors that influence the exam: punctuality, minimum equipment (computer, 
webcam and audio), good web connection 

• Addressing questions with objective answers and being prepared to interact in a short 
time in case of any initial problem (e.g. 10 minutes) 

• Being prepared to interact online by feeling calm and confident to demonstrate knowledge 
and focusing on questions not on the examiner’s physical reaction. 

3. Examiners: 

• Being able to select the most suitable format of online exams (e.g. Short e-Viva with 
tests, interactive e-Vivas without tests, long e-Vivas with presentations, ...) 

• Being prepared to interact not only through audio but also chat (e.g. copying/pasting 
questions to the chat in case of any technical problem) 

• Providing clear questions and instant feedback, and interrupting if necessary (FM option 
to stop and reinitiate the broadcasting) to keep the exam very objective and efficient. 

Limitations and future work 

To improve external validity, this research should be extended to multi-case studies at scale with 
participants of different nationalities. The FM technology has already been used in various countries 
in the last decade including multicultural and global projects, but all studies at small-scale: reflective 
learning (Wodehouse & al., 2007), peer-to-peer learning (Scott, Castaneda, Quick & Linney, 2009) 
and real-time language assessment (Hopkins, 2011). 

In this naturalistic study in Brazil, the quantitative analysis of a large metadata set of FM events 
related to four years allowed us to identify common patterns of different formats of online exams. 
The full audio recording of a representative set of events related to each format allowed us to 
develop qualitative analysis in depth as well as prepare further surveys and interviews with respective 
examiners and students. 

Our next work focuses on the use of FM in the European project ENGAGE, which offers MOOCs 
as a strategy for teachers’ continual professional development. ENGAGE includes fourteen 
Institutions in thirteen countries. Its purpose is to equip the next generation of students and 
teachers for active engagement in science through a more inquiry-based methodology and 
responsibility to come to informed decisions by reaching more than 12,000 teachers in Europe and 
360,000 students. This research in progress investigates barriers and recommendations to foster 
teacher’s communities of practice through Open Educational Resources (engagingscience.eu) 
integrated to MOOC (engage.exactls.com). The FM web videoconferencing application will be used 
in various ways for learning and assessment. 

Conclusion 

This study demonstrated that even in a challenging distributed learning context such as the one in 
Brazil, with low internet connectivity in widely distributed and isolated settlements, online video- 
conferencing technology was used effectively as a quality assurance mechanism in a scaled, low 
cost and flexible way. 

Findings in this study indicated that the main benefit was the high quality provided by these 
examinations with FM as a cost-effective alternative to F2F viva. Both examiners and students 
found e-Viva a reliable method for verifying student’s competence in writing their dissertation and 
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knowledge authorship on the field. Examiners were confidently able to rule on a student’s identity, 
knowledge, and ownership of submitted work in a short time. The average assessment length was 
seven minutes of remote video interaction. The majority of students interviewed and surveyed 
indicated various benefits of the online exam through FM: they were more focused on the questions 
instead of the physical presence of the assessor; they liked the flexibility related to location for 
assessment and found the technology easy to use. 

With regard to challenges, the key barrier was low connection in certain areas selected by students. 
This was pointed out by both examiners and students. However, all of these cases were solved by 
rebooking the online exam in a different venue. 

This large-scale study has clearly demonstrated the advantages of a well-designed use of the 
e-Viva at scale to complement a distance study programme, as well as provided some key 
recommendations for institutions, examiners and students. 

This event design did not require any significant investment in human proctoring. Student identity 
was easily checked via video which was recorded, and ‘cheating’ was very unlikely in this context, 
so the exam was both demonstrably of good quality and well standardized. Our next work will focus 
on a multicultural context of FM applied to assessment in multilingual MOOCs. 
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